Apparatus and method for dynamic graphics rendering based on saccade detection

ABSTRACT

A method for rendering computer graphics based on saccade detection is provided. One embodiment of the method includes rendering a computer simulated scene for display to a user, detecting an onset of a saccade that causes saccadic masking in an eye movement of the user viewing the computer simulated scene, and reducing a computing resource used for rendering frames of the computer simulated scene during at least a portion of a duration of the saccade. Systems perform similar steps, and non-transitory computer readable storage mediums each storing one or more computer programs are also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/845,862, filed on Sep. 4, 2015, entitled “APPARATUS AND METHOD FORDYNAMIC GRAPHICS RENDERING BASED ON SACCADE DETECTION,” the entirecontent and disclosure of which is hereby fully incorporated byreference herein in its entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates generally to computer generated images,and more specifically to real-time rendering of computer generatedgraphics.

2. Discussion of the Related Art

Computer rendered graphics are used to present various user interfacesand media to users. Real-time rendering of computer graphics allows thegraphical content to be rendered in response to user inputs and softwarestate changes in real-time.

SUMMARY OF THE INVENTION

One embodiment provides a method comprising: rendering a computersimulated scene for display to a user, detecting an onset of a saccadethat causes saccadic masking in an eye movement of the user viewing thecomputer simulated scene, and reducing a computing resource used forrendering frames of the computer simulated scene during at least aportion of a duration of the saccade.

Another embodiment provides a system comprising: an eye movement sensor,a display device for displaying computer simulated scenes to a user, anda processor communicatively coupled to the eye movement sensor and thedisplay device. The processor being configured to render a computersimulated scene for display on the display device, detect an onset of asaccade that causes saccadic masking in an eye movement of the userviewing the computer simulated scene based on signals from the eyemovement sensor, and reduce a computing resource used for renderingframes of the computer simulated scene during at least a portion of aduration of the saccade.

Another embodiment provides a non-transitory computer readable storagemedium storing one or more computer programs configured to cause aprocessor based system to execute steps comprising: rendering a computersimulated scene for display to a user, detecting an onset of a saccadethat causes saccadic masking in an eye movement of the user viewing thecomputer simulated scene, and reducing a computing resource used forrendering frames of the computer simulated scene during at least aportion of a duration of the saccade.

A better understanding of the features and advantages of variousembodiments of the present invention will be obtained by reference tothe following detailed description and accompanying drawings which setforth an illustrative embodiment in which principles of embodiments ofthe invention are utilized.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of embodiments ofthe present invention will be more apparent from the following moreparticular description thereof, presented in conjunction with thefollowing drawings wherein:

FIG. 1 is a block diagram illustrating a system for dynamic graphicsrendering based on saccade detection in accordance with some embodimentsof the present invention;

FIG. 2 is a flow diagram illustrating a method for dynamic graphicsrendering based on saccade detection in accordance with some embodimentsof the present invention;

FIG. 3 is an illustration of computing resource reduction with saccadedetection in accordance with some embodiments of the present invention;

FIG. 4 is a block diagram illustrating a calibration process inaccordance with some embodiments of the present invention;

FIGS. 5A-5B are illustrations of head mounted devices in accordance withsome embodiments of the present invention.

DETAILED DESCRIPTION

The human visual system has an eye motion called a saccade that is arapid movement from one gaze position to another. During a saccade, thebrain masks the signals that it receives from the optic nerve. Thismasking effect is generally referred to as saccadic masking. As soon asthe saccade is over and the eyes have reached a stable new fixationpoint, the image that is sent via the optic nerve to the brain is thenused to “fill in” the perceptual gap during the saccade. Otherperceptual effects such as spatial updating and trans-saccadicintegration also affect how the human brain perceive images around asaccade event. Spatial updating refers to the effect that a visualstimulus viewed prior to a saccade is still perceived by the brainduring and after a saccade even when the visual stimulus is no longervisible. Tran-saccadic integration refers to the effect that the brainintegrates visual information from two or more fixation points aroundsaccades.

The saccade masking time can sometimes begin before the saccadic eyemotion by some number of milliseconds (such as approximately 30 ms). Theeye movement itself can last approximately 50-75 ms depending upon themagnitude of the saccade. After the eye motion stops, anotherapproximately 30 ms of brain signal masking may occur.

In computer rendered virtual reality (VR) scenes, real-time imagerendering takes a lot of central processing unit (CPU) and graphicprocessing unit (GPU) power and images are generated frequently(generally at least 60 Hz) to give a good VR perception. Therefore,rendering for VR can be very power consuming.

During a saccade, new images (frames) may not be perceived by a user dueto saccadic masking. A saccadic masking period may easily be around 100ms to 200 ms in some cases. A system that knows that a saccade maskingperiod is occurring can avoid rendering frames of images to savecomputing resources and power consumption. For example, at 60 Hzrendering and a saccadic period of 100 ms, approximately 6 frames can beskipped or rendered with less computing resources during the saccade. Inmany VR systems, higher frame rates such as 90 Hz or 120 Hz are used. Insuch systems, more frames could be saved during a typical saccadicperiod. The total power saving of applying this technique may be relatedto the percentage of total VR viewing time that is taken by saccadicmasking periods.

In addition to power saving, in some embodiments, the “saved” processingtime may be used to render low frequency updates such as complexlighting calculations. Typically, scene lighting in 3D is relativelyslow changing and so can be calculated with greater tolerance forlatency. Since saccades are not guaranteed to occur at any specificrate, this technique is suited for long duration (several seconds)calculations where there is a statistical likelihood of getting asaccade. This technique may be used in conjunction with foveal renderingand gaze tracking.

Various techniques for detecting saccades will be discussed below. Forexample, saccades can be detected optically using a high rate sensorsuch as a position sensitive detector (PSD) or a Doppler Effect sensor.In some embodiments, the onset of a saccade may be detected by measuringelectromyography (EMG) muscles signals on the skin close to the eyemuscles.

A more detailed description will now be provided. Referring first toFIG. 1, there is shown a system for rendering images based on saccadedetection that may be used to run, implement and/or execute any of themethods and techniques shown and described herein in accordance withsome embodiments of the present invention. The saccade detection system100 includes a processor 101, a memory 102, a display 130, and a sensor110. The saccade detection system 100 may comprise, or be included with,one or more of a head mounted display (“HMD”) device, an augmentedreality device, a virtual reality device, a wearable device, a portableuser device, a smartphone, a personal computer, a television, a tabletcomputer, a game console, etc. Generally, the saccade detection system100 may comprise any processor-based device having at least one sensor.

The memory 102 may include one or more of a volatile and/or non-volatilecomputer readable memory devices. In some embodiments, the memory 102stores computer executable code that causes the processor 101 toautomatically detect an onset of a saccade and reduce the computingresource used for rendering frames of a computer simulated scene duringat least a portion of a duration of the saccade. The computer readableinstructions may instruct the processor to detect a saccade based oninformation from the sensor 110. The memory 102 may further store an eyemovement model used to determine whether the detected eye movementconstitutes a saccade that causes saccadic masking and/or to predict aduration of the detected saccade. The eye movement model may includeparameters calibrated for individual users through one or more of thecalibration processes described herein. In some embodiments, the memory102 further stores computer executable code that causes the processor101 to provide interactive audio and video content to the user. Forexample, the interactive video content may be virtual reality oraugmented reality content. Video content may comprise computer generatedimages rendered in real-time. In some embodiments, computer executablecode causes the processor 101 to perform one or more steps describedherein with reference to FIGS. 2 and 4 below. In some embodiments, thememory 102 may be at least partially implemented by one or more of alocal, remote, and cloud-based storage.

The sensor 110 may be or comprise one or more of a camera, an opticalsensor, an infrared sensor, an EMG sensor, an optical reflector sensor,a range sensor, an optical flow sensor, a Doppler sensor, a microphoneand the like. Generally, the sensor 110 is configured to detect a rapideye movement such as a change in eye movement direction, acceleration,and speed. Eye movement may be tracked based on a series of detected eyegaze positions relative to the frame of the captured image and/or thehead of the user. For example, the sensor 110 may capture a series ofimages of the pupil of an eye, and each a gaze location may bedetermined based on the location of the center of the pupil within thecaptured frame.

An EMG sensor is a sensor that detects the electrical potentialgenerated by muscle cells. An EMG placed on or in proximity of theorbital or periorbital region of an eye can measure the amplitude and/ordirection of eye movement based on electrical potential generated bymuscles that control eye movement (e.g. the lateral rectus, the medialrectus, the inferior rectus, the superior rectus, the inferior obliquemuscles, etc.). In some embodiments, the EMG sensor may be placeddirectly on the skin of the user over the muscle that the sensor isconfigured to measure. In some embodiments, the EMG sensor does notcontact the skin of the user. In some embodiments, an EMG may detect anonset of a saccade based on muscle signals prior to the start ofsaccadic movement of the eye.

An optical reflector sensor may be a sensor that detects eye movement bydetecting changes in light reflected off of the eyeball. For example, anoptical reflector sensor may work in a similar manner as an opticaltrack ball device. A range sensor may be any sensor configured to detectthe presence of nearby objects without physical contact, such as aDoppler sensor, a passive optical sensor, an infrared sensor, a radar,and the like. Since a human eyeball is not perfectly spherical, theproximity between a sensor at a fixed distance from the skull of a userand the portion of the eyeball in the sensor's direct line-of-sightchanges with eye movement. For example, the cornea of an eye is raisedrelative to the sclera, therefore, a shorter detected range may indicatethat the cornea is in the sensor's direct line-of-sight. A microphonemay be used to detect audio signals produced by eye movement. Forexample, amplitude of the sound produced by an eye movement maycorrespond to the amplitude of the eye movement. Directional eyemovement may also be detected based on their respective sound profile.

The display 130 may be or comprise one or more of a display screen, aprojection device, an augmented reality display device, a virtualreality display device, a HMD, and the like. Generally, the display 130is configured to show computer generated graphics from the processor 101to a user. While the display 130 is shown to be part of the saccadedetection system 100, in some embodiments, the display 130 may beseparately implemented from the saccade detection system 100. Forexample, the processor may output the generated graphics to an externaldisplay device to display to a user. The saccade detection system 100may monitor a user's eye movement as the user views the computergenerated graphics shown by the display 130 and/or real-world scenes.

In some embodiments, the saccade detection system 100 further includes agraphic processing unit (GPU) that may be integrated with or discretefrom the processor 101. The GPU may perform at least some of therendering calculations in generating graphics to display to a user.Generally, computing resources may refer to either or both of CPU andGPU processing resources.

In some embodiments, the saccade detection system 100 further includes aphysical structure that holds, supports, and/or mounts the sensor 110 ina position suitable for sensing the user's eye movement. For example, insome embodiments, the physical structure may hold a camera slightly infront the user's eye or an EMG sensor directly on or in close proximityof the outer corner of the user's eye. Examples of such physicalstructures according to some embodiments are provided in FIGS. 5A-5Bbelow. In some embodiments, only one eye is monitored, while in otherembodiments, the system monitors both eyes with sets of sensors. In someembodiments, one or more of the sensors comprise multiple sensor and/orsensor types.

In some embodiments, the user's eye gaze may be tracked by the sensor110 or a second sensor (not shown). For example, the system may use thesensor 110 to track the user's eye gaze positions as an input to a VR orAR system and also use the eye tracking data for saccade detection. Insome embodiments, the user's eye gaze may be tracked by a camera and/oran EMG sensor, and the saccade detection is based on only the EMG sensoror another sensor not used for gaze tracking. The system may furtherinclude a sensor for detecting the end of a saccade. By way of example,the sensor for detecting the end of the saccade may be or comprisesensor 110, a sensor used for tracking eye gaze, and/or a separatesensor.

In some embodiments, the saccade detection system 100 may include otherinput/output devices such as speakers, audio output ports, keys, touchpads, touch screens, microphones, gyroscopes, wireless transceivers, andthe like. In some embodiments, one or more methods and functionsdescribed herein may be performed by a remote device and communicated tothe saccade detection system 100 via a wired or wireless dataconnection. In some embodiments, the processor 101 is configured to usethe tracked eye movement and data received from a remote source todetermine the content to display to the user on the display 130. Forexample, the processor 101 may cause the display 130 to display local orexternal content in an augmented reality or virtual reality manner basedon the user's tracked eye movement. The processor 101 may furtherdetermine whether and how to reduce the computing resources used forrendering frames of the content based on detected saccades.

In some embodiments, each component shown in FIG. 1 is enclosed and/oraffixed to a portable housing such as a head mounted device (HMD) or awearable device. In some embodiments, one or more components shown inFIG. 1 may be separately implemented and communicate with the systemthrough a wired or wireless connection. For example, the sensor 110 maybe a sensor placed near a computer monitor or television set, and thememory 102 and the processor 101 shown in FIG. 1 may be implemented witha personal computer system, a game console, or an entertainment system.

Referring to FIG. 2, there is illustrated an example of a method 200 forrendering images based on saccade detection. In some embodiments, stepsof method 200 may be performed by one or more server devices, userdevices, or a combination of server and user devices. Generally, thesteps of method 200 may be performed by one or more processor-baseddevices such the processor 101 of the saccade detection system 100and/or other control circuits.

In step 210, the system renders a computer simulated scene for displayto a user. In some embodiments, the computer simulated scene is renderedin real-time based on user inputs and/or program states. In someembodiments, the computer simulated scene is rendered at a constantframe rate such as 60 Hz, 90 Hz, 120 Hz, etc. In some embodiments, thecomputer simulated scene is rendered at a variable frame rate. Thecomputer simulated scene may comprise one or more of a VR scene, an ARscene, a 3D scene, a 2D scene, and the like. The rendering of thecomputer simulated scene may include the rendering of 2D objects, 3Dobjects, backgrounds, shadows, lighting effects, user interfaceelements, and the like. In some embodiments, the computer simulatedscene may be part of an interactive movie, a video game, a simulatorsoftware, and the like.

In step 220, the system detects an onset of saccade that causes saccadicmasking. A saccade generally refers to a quick movement of eyes betweentwo phases of fixation. A saccade is distinguished from other types ofeye movement such as ocular tremor, ocular drift, and smooth pursuit. Asaccade of sufficient magnitude causes saccadic masking in brainperception. The system may distinguish a saccade that causes saccadicmasking from microsaccades, which are small involuntary eye movement. Asaccade may be detected by detecting a sudden change in one or more ofdirection, speed, and acceleration of eye movement. In some embodiments,an EMG sensor may be used to monitor for muscle signals associated withan onset of a saccade. For example, a saccade may be detected bydetecting a spike in the electrical potential generated by one or moremuscles that control eye movement. In some embodiments, a saccade isdetected when an acceleration of eye movement that exceeds apredetermined threshold is detected. In some embodiments, the detectionof saccade may be based on an eye movement model. For example, a thethreshold of eye movement acceleration used for detecting saccade may bebased on a user's demographic or may be calibrated for the individualuser using one or more calibration methods described herein. Generally,any known method of saccade detection may be used without departing fromthe spirit of the present disclosure.

In some embodiments, a duration of the saccade is predicted after step220. In some embodiments, the duration of the saccade may be predictedbased on at least one of an eye movement model and a magnitude of ameasurement used to detect the onset of the saccade. The eye movementmodel may be based on statistical data relating to durations of saccadesin human eyes and/or be calibrated for the individual user. Statisticaldata relating to durations of saccades in human eyes may be based on asurvey of the general population that provides information on thetypical saccade durations of most people. In some embodiments, theduration of the saccade is predicted based on the magnitude of thesaccade at the onset. The duration of a saccade may generally correlateto the magnitude of eye movement acceleration at the onset. Therefore,the system may measure the data relating to the acceleration at theonset to predict the duration of the saccade. In some embodiments, anEMG sensor measures electrical potential generated by muscles thatcontrol eye movement and the prediction of the duration of the saccadeis based on the value of the measured electrical potential. In someembodiments, the speed of eye movement may be calculated from at leasttwo images captured by an image sensor to detect a saccade. The durationof the saccade may be predicted based on the speed of eye movementshortly after the onset of a saccade. In some embodiments, the systemmonitors for a peak eye movement speed and/or eye movement decelerationduring a saccade and predicts the end of the saccade based on thedetected peak eye movement speed and/or eye movement deceleration.

In step 230, the system reduces computing resource used for renderingframes of the computer simulated scene during at least a portion of aduration of the saccade. In some embodiments, the system may reduce thecomputing resource used for rendering frames by skipping one or moreframes of the computer simulated scene, repeating one or more frames ofthe computer simulated scene, reducing a rendering detail of one or moreframes of the computer simulated scene, reducing a rendering resolutionof one or more frames of the computer simulated scene, and/or adjustinga variable frame rate for rendering the computer simulated scene. Insome embodiments, the system repeats the frame generated just before thereduction period to avoid performing calculations for the frames duringthe reduction period. In a system with variable frame rate rendering,such as a system with a variable frame rate GPU, the system may pauserendering until the end of the saccadic masking period or render at alower frame rate during the reduction period. In some embodiments, thesystem reduces the rendering details by reducing the details in thetexture map, reducing the polygon count of the 3D models, and reducinglighting effects such as shadows, reflections, and subsurfacescattering. In some embodiments, the frames in the reduction period maybe generated with a technique similar to the reprojection technique ingroup of pictures (GOP) video encoding. With bidirectional projection, aB frame references images preceding and following that frame, andcontains motion-compensated difference information relative to thereferenced frames.

Generally, during the reduction period, the system renders the computersimulated scene in a way that requires fewer calculations by the CPUand/or the GPU of the system. Due to saccadic masking, spatial updating,and/or trans-saccadic perception, frames, and/or elements and details ofa frame may be omitted without being perceptible to a viewer.

In some embodiments, the reduction of computing resources used forrendering frames lasts for at least a portion of the predicted durationof a saccade. For example, with a 60 Hz rendering, if a saccade ispredicted to last 100 ms, the reduction may last 1-6 frames. In someembodiments, the system may be configured to leave a “safe” zone at thebeginning and/or the end of a detected saccade period in which theframes are rendered as normal. In some embodiments, the system may takein to effect that saccadic masking may persist for some time after theend of saccadic movement of the eye and allow the reduction period toexceed the expected duration of saccadic eye movement. In someembodiments, the system does not predict the duration of the saccade andthe reduction period may be set to a default minimum length for alldetected saccades. For example, if only saccades having an onsetmagnitude that typically corresponds to at least 100 ms of duration aredetected in step 220, the reduction period may be set to 90 ms for alldetected saccades without the system predicting a duration for eachdetected saccade.

In some embodiments, in step 220, the system also determines aconfidence level of the detection of the onset of the saccade. Theconfidence level may be based on the clarity of the measured signaland/or the magnitude of the change in one or more of eye movementdirection, acceleration, and speed. In some embodiments, the systemdetermines the amount of computing resources to reduce based on theconfidence level. The amount of computing resources to reduce may bebased on how many frames to skip, how many frames to repeat, how muchrendering detail to omit, the rendering resolution, the rendering framerate, etc. For example, with a high confidence level, the renderingresolution may be dropped to a fourth of the original renderingresolution; while with a low confidence level, the rendering resolutionmay be dropped to half of the original rendering resolution. In someembodiments, the system selects from two or more methods of reducing thecomputing resources for rendering frames of the computer simulated scenebased on the confidence level. For example, with a high confidencelevel, the system may skip or repeat frames; while with a low confidencelevel, the system may reduce the detail of the rendered frame.

In some embodiments, the amount and the method of reduction maycorrespond to the temporal location of a frame within a saccade period.For example, less reduction may be made to frames nearer to thebeginning and the end of a saccade period as compared to frames in themiddle of a saccade period.

In some embodiments, the system is further configured to reallocate thecomputing resources saved in step 230 to perform low frequency updatecalculations for rendering the computer simulated scene. For example,the system may reallocate the computing resource to perform complexlighting calculations for the computer simulated scene. In someembodiments, the freed computing resource may be used by otherapplication and system processes.

Since each individual's physiology varies, in some embodiments, prior tothe process shown in FIG. 2, the system may lead a user through acalibration sequence to generate or configure an eye movement modelindividualized for the user. For example, the system may instruct a userto move their eyes from one angular position to another. The system maythen compare the measurements obtained by the sensor to the expected eyemovement to configure the eye movement model for that user. The eyemovement model may then be used to detect a saccade that causes saccadicmasking and/or predict the duration of the saccade.

In some embodiments, in addition to or instead of performing acalibration sequence, after step 220, the system may detect an end of asaccade period for calibration purposes. The end of a saccade period maybe detected by eye tracking methods and may correspond to a return to aneye movement speed or pattern corresponding to other types of eye motionstates such as fixation, ocular tremor, ocular drift, microsaccade, andsmooth pursuit. The end of a saccade period may be detected by the sameor a different sensor used to detect the onset of the saccade in step220. The measured duration of a saccade period may be compared to thepredicted duration of the saccade to calibrate the eye movement modelused to predict the duration of the saccade. A more detailed descriptionof the calibration process is provided herein with reference to FIG. 4below.

In some embodiments, the saccade detection in FIG. 2 may be used inconjunction with foveated rendering. Foveated rendering is a techniquein which the image resolution or amount of detail varies across theimage according to fixation points. When a saccade is detected, thesystem may assumed no fixation point during the duration of the saccadeand reduce the image resolution or the amount of detail in the renderedimage to a baseline value across the entire image.

Referring to FIG. 3, there is shown an illustration of reducingcomputing resource with saccade detection in accordance with someembodiments of the present invention. In FIG. 3, eye velocity oracceleration may be tracked by a sensor such as sensor 110 in FIG. 1.When eye velocity or acceleration exceeds a threshold, the system maydetect a saccade onset 310. In some embodiments, the onset may bedetected based on muscle signals prior to the increase of eye velocityor acceleration. In some embodiments, the onset of a saccade may bedetected shortly after the actual beginning of the saccadic movement.

In some embodiments, the system also predicts a duration of the saccade312 based on measurements at the onset of the saccade. In someembodiments, the prediction may be based on the magnitude of theacceleration at the saccade onset 310. When a saccade of a magnitudethat causes saccadic masking is detected, the system may reduce thecomputing resource used for rendering frames by skipping one or moreframes of the computer simulated scene, repeating one or more frames ofthe computer simulated scene, reducing a rendering detail of one or moreframes of the computer simulated scene, reducing a rendering resolutionof one or more frames of the computer simulated scene, and/or adjustinga variable frame rate for rendering the computer simulated scene. Asshown in FIG. 3, the computing resource used for rendering frames of thecomputer simulated scene drops during a portion of the predictedduration of saccade. The computing resource line generally represent theload on the CPU and/or GPU for rendering the images. For example, thecomputing resource line may represent the number of calculationsperformed per millisecond by the CPU/GPU. The system returns therendering method back to the normal baseline just prior to the predictedend of a saccade period. The predicted end of a saccade period maycorrespond to the predicted end of saccadic movement or the predictedend of the saccadic masking perceptual effect, which could last for sometime after the end of the saccadic movement.

FIG. 3 is provided as illustrations of concepts only. FIG. 3 does notcorrespond to actual saccade detection or computing resource reductionand is not necessarily to scale.

The reduction in computing resources during saccade periods can reducethe overall power consumption of the rendering device. For portabledevices using batteries, the reduction of power consumption can lead toincreased battery life between charges. The computing resources may alsobe allocated for other uses such as high latency updates for renderingthe computer simulated scenes. The computing resources may also be freedup for use by other application and/or the operating system processesrunning on the device.

Referring to FIG. 4, there is illustrated an example of a saccadeduration prediction with feedback calibration. The sensor 420 isconfigured to measure one or more of eye movement direction, speed, andacceleration and provide the measurement to a saccade detection module422. The sensor 420 may be the sensor 110 shown in FIG. 1 and/or maycomprise one or more of a camera, an optical sensor, an optical imagesensor, an infrared sensor, an optical flow sensor, an EMG sensor, aposition sensing device, a Doppler effect sensor, and the like.

The saccade detection module 422 analyzes the data from the sensor 420to determine whether a saccade that causes saccadic masking is occurringor is about to occur. In some embodiments, saccade detection module 422may base its determination on the eye movement model 450 or a separatemodel. For example, an eye movement model may include an eye movementacceleration magnitude threshold indicative of an onset of a saccadethat causes saccadic masking. For embodiments with an EMG sensor, thethreshold may correspond to a muscle signal voltage.

After an onset of a saccade is detected, the saccade duration predictionmodule 425 predicts the duration of the saccade. In some embodiments,the saccade duration prediction module 425 bases its prediction on oneor more of the magnitude of eye movement at the onset of the saccade anda detected peak acceleration or speed of the saccade. The saccadeduration prediction module 425 further bases its prediction on an eyemovement model 450. The eye movement model 450 may correspond detectedsaccade magnitudes to saccade durations. The eye movement model 450 mayinitially be based on statistical data relating to durations of saccadesin human eyes and later be calibrated for the individual user.

Also after an onset of a saccade is detected, the saccade durationmeasurement module 430 monitors for the end of the saccade period andcalculates the duration of the saccade. The end of the saccade periodmay be detected by eye tracking methods and may correspond to a returnof eye movement speed or pattern corresponding to other types of eyemotion states such as fixation, ocular tremor, ocular drift,microsaccade, and smooth pursuit. The end of a saccade period may bedetected by the sensor 420 and/or one or more other sensors. Thecalibration module 440 compares the predicted saccade duration from thesaccade duration prediction module 425 and the measured saccade durationfrom the saccade duration measurement module 430 to determine theaccuracy of the prediction. The calibration module 440 may update theeye movement model 450 based on the comparison. For example, if thesaccade duration prediction module 425 tends to underestimate theduration of a saccade, the calibration module 440 may update the eyemovement model 450 to correspond measurements at the onset of a saccadewith a longer saccade duration.

While the eye movement model 450 is shown to be used by the saccadeduration prediction module 425 in FIG. 4, in some embodiments, thesaccade detection module 422 also uses the same or a different eyemovement model 450. In some embodiments, the calibration module 440 maybase its calibration on multiple saccade duration predictions andsaccade duration measurements, and only update the eye movement model450 when a trend is observed.

Referring to FIG. 5A, there is shown an illustration of an augmentedreality type HMD with which the technique disclosed herein may beimplemented. The HMD device 500 includes a frame 510 holding sensors 513and 515 in positions suitable for monitoring a wearer's eye movement.While two sensors are shown in FIG. 5A, the techniques described hereinmay be implemented with HMDs having only the first sensor 513 or thesecond sensor 515.

In some embodiments, the HMD device 500 includes two sensors; one sensoris used to detect the onset of a saccade and another sensor is used tomeasurement the duration of a saccade. In some embodiments, one sensoris used to detect both the onset of the saccade and the end of thesaccade. In some embodiments, the HMD does not include a sensor fordetecting the end of a saccade. The HMD device 500 also includes adisplay device (not shown) that is configured to provide an augmentedreality scene to the user. The first sensor 513 may be an optical sensorthat is positioned to capture images of the user's eye. The secondsensor 515 may be an EMG sensor that is either contacting or in closeproximity of the temple region of the user as shown in FIG. 5A. An EMGsensor may be placed in other positions, for example, below the user'seye, near the inner corner of the eye, etc. In some embodiments, two ormore EMG sensors may be placed in different areas. The second sensor 515may comprise other types of sensors such as a low resolution imagesensor, a high frame-rate image sensor, an optical reflector sensor, arange sensor, and a microphone. In embodiments in which the secondsensor is an image or optical sensor, the second sensor 515 may beplaced near the first sensor 513. The placement of sensors in FIG. 5Aare provided as an example only. Generally, the placement of the sensorsmay be configured variously based on sensor type and the user'sphysiology without departing form the spirit of the present disclosure.

Referring to FIG. 5B, there is shown an illustration of a virtualreality type HMD device with which the technique disclosed herein may beimplemented. The HMD device 520 encloses the field of vision of theuser. Sensor(s) described herein may be placed on the user facing sideof the HMD. For example, an image sensor may be positioned just abovethe display screen inside the HMD device 520 and an EMG sensor may bepositioned on a portion of the HMD device 520 resting against theorbital or periorbital areas of the user.

In both FIGS. 5A-5B, detected saccades may be used by the HMD devices500 and 520 to determine how to render computer generated graphics todisplay to the user. When a saccade is detected, the processor of theHMD devices 500 and 520 may reduce the computing resources used forrendering frames of the computer simulated scene. The determined andestimated gaze locations may also be used to predict a future gazelocation which allows the computer to determine the fixation point ofthe user when the next frame is displayed to facilitate foveatedrendering.

In some embodiments, one or more of the embodiments, methods,approaches, and/or techniques described above may be implemented in oneor more computer programs or software applications executable by aprocessor based apparatus or system. By way of example, such processorbased apparatuses or systems may comprise a computer, entertainmentsystem, game console, workstation, graphics workstation, server, client,portable device, pad-like device, etc. Such computer program(s) may beused for executing various steps and/or features of the above-describedmethods and/or techniques. That is, the computer program(s) may beadapted to cause or configure a processor based apparatus or system toexecute and achieve the functions described above. For example, suchcomputer program(s) may be used for implementing any embodiment of theabove-described methods, steps, techniques, or features. As anotherexample, such computer program(s) may be used for implementing any typeof tool or similar utility that uses any one or more of the abovedescribed embodiments, methods, approaches, and/or techniques. In someembodiments, program code macros, modules, loops, subroutines, calls,etc., within or without the computer program(s) may be used forexecuting various steps and/or features of the above-described methodsand/or techniques. In some embodiments, the computer program(s) may bestored or embodied on a computer readable storage or recording medium ormedia, such as any of the computer readable storage or recording mediumor media described herein.

Therefore, in some embodiments the present invention provides a computerprogram product comprising a medium for embodying a computer program forinput to a computer and a computer program embodied in the medium forcausing the computer to perform or execute steps comprising any one ormore of the steps involved in any one or more of the embodiments,methods, approaches, and/or techniques described herein. For example, insome embodiments the present invention provides one or morenon-transitory computer readable storage mediums storing one or morecomputer programs adapted or configured to cause a processor basedapparatus or system to execute steps comprising: rendering a computersimulated scene for display to a user, detecting an onset of a saccadethat causes saccadic masking in an eye movement of the user viewing thecomputer simulated scene, and reducing a computing resource used forrendering frames of the computer simulated scene during at least aportion of a duration of the saccade.

While the invention herein disclosed has been described by means ofspecific embodiments and applications thereof, numerous modificationsand variations could be made thereto by those skilled in the art withoutdeparting from the scope of the invention set forth in the claims.

1-20. (canceled)
 21. A method, comprising: detecting a saccade in an eye of a user viewing a computer simulation; predicting, prior to an end of the saccade, where a fixation point of the user will be located in a computer simulated scene at the end of the saccade; and rendering the predicted fixation point in a greater amount of detail than other portions of the computer simulated scene.
 22. The method of claim 21, wherein the rendering the predicted fixation point in a greater amount of detail than other portions of the computer simulated scene comprises: rendering the predicted fixation point at a higher resolution than other portions of the computer simulated scene.
 23. The method of claim 21, wherein the predicting where a fixation point of the user will be located in a computer simulated scene at the end of the saccade comprises: predicting a future gaze location of the user.
 24. The method of claim 21, wherein the predicting where a fixation point of the user will be located in a computer simulated scene at the end of the saccade comprises: predicting a duration of the saccade.
 25. The method of claim 24, wherein the predicting the duration of the saccade is based on at least one of statistical data relating to durations of saccades in human eyes and a magnitude of a measurement used to detect the onset of the saccade.
 26. The method of claim 24, further comprising: detecting the end of the saccade; and calculating the duration of the saccade based on at least the detected end of the saccade.
 27. The method of claim 26, further comprising: comparing the calculated duration of the saccade to the predicted duration of the saccade; and using results of the comparison to adjust a model used to predict the duration of the saccade.
 28. The method of claim 21, wherein the saccade is detected using one or more of an optical sensor, an optical flow sensor, an electromyography (EMG) sensor, a position sensing device, and a Doppler effect sensor.
 29. The method of claim 21, wherein the detecting a saccade in an eye of a user viewing a computer simulation comprises: detecting an onset of the saccade prior to a start of saccadic movement of the eye.
 30. The method of claim 21, further comprising: calibrating an individualized eye movement model for the user viewing the computer simulation, wherein the detecting the saccade is based on the individualized eye movement model.
 31. The method of claim 21, further comprising: reducing a computing resource used for rendering frames of the computer simulation during at least a portion of a duration of the saccade.
 32. The method of claim 31, further comprising: determining a confidence level of the detection of the saccade.
 33. The method of claim 32, further comprising: determining an amount of the computing resource to reduce based on the confidence level.
 34. The method of claim 32, further comprising: selecting from two or more methods of reducing the computing resource used for rendering frames of the computer simulation based on the confidence level.
 35. The method of claim 31, further comprising: reallocating the computing resource to perform low frequency update calculations for rendering frames of the computer simulation.
 36. The method of claim 35, wherein the low frequency update calculations comprise lighting calculations.
 37. A system comprising: an eye movement sensor; a display device for displaying computer simulated scenes to a user; and a processor communicatively coupled to the eye movement sensor and the display device, the processor being configured to execute steps comprising: detecting a saccade in an eye of the user viewing the computer simulated scenes; predicting, prior to an end of the saccade, where a fixation point of the user will be located in a computer simulated scene at the end of the saccade; and rendering the predicted fixation point in a greater amount of detail than other portions of the computer simulated scene.
 38. The system of claim 37, wherein the rendering the predicted fixation point in a greater amount of detail than other portions of the computer simulated scene comprises: rendering the predicted fixation point at a higher resolution than other portions of the computer simulated scene.
 39. The system of claim 37, wherein the predicting where a fixation point of the user will be located in a computer simulated scene at the end of the saccade comprises: predicting a future gaze location of the user.
 40. The system of claim 37, wherein the predicting where a fixation point of the user will be located in a computer simulated scene at the end of the saccade comprises: predicting a duration of the saccade.
 41. The system of claim 37, wherein the eye movement sensor comprises one or more of an optical sensor, an optical flow sensor, an electromyography (EMG) sensor, a position sensing device, and a Doppler effect sensor.
 42. The system of claim 37, wherein the detecting a saccade in an eye of the user viewing the computer simulated scenes comprises: detecting an onset of the saccade prior to a start of saccadic movement of the eye.
 43. The system of claim 37, wherein the processor is further configured to execute steps comprising: calibrating an individualized eye movement model for the user viewing the computer simulated scenes, wherein the detecting the saccade is based on the individualized eye movement model.
 44. The system of claim 37, wherein the processor is further configured to execute steps comprising: reducing a computing resource used for rendering frames of the computer simulated scenes during at least a portion of a duration of the saccade.
 45. The system of claim 44, wherein the processor is further configured to execute steps comprising: determining a confidence level of the detection of the saccade.
 46. The system of claim 45, wherein the processor is further configured to execute steps comprising: determining an amount of the computing resource to reduce based on the confidence level.
 47. The system of claim 45, wherein the processor is further configured to execute steps comprising: selecting from two or more methods of reducing the computing resource used for rendering frames of the computer simulated scenes based on the confidence level.
 48. A non-transitory computer readable storage medium storing one or more programs that cause a processor based system to execute steps comprising: detecting a saccade in an eye of a user viewing a computer simulation; predicting, prior to an end of the saccade, where a fixation point of the user will be located in a computer simulated scene at the end of the saccade; and rendering the predicted fixation point in a greater amount of detail than other portions of the computer simulated scene.
 49. The non-transitory computer readable storage medium of claim 48, wherein the rendering the predicted fixation point in a greater amount of detail than other portions of the computer simulated scene comprises: rendering the predicted fixation point at a higher resolution than other portions of the computer simulated scene.
 50. The non-transitory computer readable storage medium of claim 48, wherein the predicting where a fixation point of the user will be located in a computer simulated scene at the end of the saccade comprises: predicting a future gaze location of the user.
 51. The non-transitory computer readable storage medium of claim 48, wherein the predicting where a fixation point of the user will be located in a computer simulated scene at the end of the saccade comprises: predicting a duration of the saccade.
 52. The non-transitory computer readable storage medium of claim 48, wherein the detecting a saccade in an eye of a user viewing a computer simulation comprises: detecting an onset of the saccade prior to a start of saccadic movement of the eye.
 53. The non-transitory computer readable storage medium of claim 48, wherein the one or more programs are further configured to cause the processor based system to execute steps comprising: reducing a computing resource used for rendering frames of the computer simulation during at least a portion of a duration of the saccade.
 54. The non-transitory computer readable storage medium of claim 53, wherein the one or more programs are further configured to cause the processor based system to execute steps comprising: determining a confidence level of the detection of the saccade.
 55. The non-transitory computer readable storage medium of claim 54, wherein the one or more programs are further configured to cause the processor based system to execute steps comprising: selecting from two or more methods of reducing the computing resource used for rendering frames of the computer simulation based on the confidence level. 